Machine and Human Performance for Single and Multidocument Summarization

coherency—and be able to draw the “best” information from a set of documents. Automatic single-document text summarization1 has been an active research area since the 1950s, with a renaissance of approaches since the 1990s. Human single-document summarization is well defined when guidelines and recommendations drive performance.2,3 System-generated single-document summaries, while not always matching well with reference summaries, are generally good quality and have proved useful. In contrast to the single-document task, multiple-document summarization development (automated or not) lacks complementary documentation of procedures and methodologies for human performance. Although researchers have explored various strategies for analyzing documents in a collection and then synthesizing and condensing information to produce multidocument summaries, they have not yet seen strong system performance. The lack of guidelines has a greater impact on multidocument summarization than on the single-document task. Analyzing and synthesizing such extensive information tax both human and machineprocessing capabilities. The recent NIST-sponsored (National Institute of Standards and Technology) Document Understanding Conference II (DUC 2002)4 evaluated automatic multidocument summarization capability using 59 single-document collections with an average of 10 documents per set. These sets covered single events, multiple related events, and biographies. NIST provided 30 document sets, including single documents, single-document human-generated reference summaries, and multidocument human-generated reference summaries for each set. Additional multidocument summaries for each set, which different human summarizers wrote, were also available, as was data from the DUC 2001 conference. For evaluation, human evaluators compared system-generated summaries to the reference human summaries. Our prototype multidocument summarizer operates by first generating single-document summaries and then selecting from those sentences to produce the multidocument summary. It is based on our current text summarization system, which delivers, in real time, indicative summaries for a high-volume, heterogeneous document collection to US government users. Thus, the constraints of our environment tend to prohibit the use of more knowledge-intense approaches but truly represent typical requirements for commercial viability. We believe our environment highlights actual practical performance demands and NLP challenges.